Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffAaron Ximm Date: Jul 25, 2012 5:21pm
Forum: etree Subject: Internet Archive adds BitTorrents for downloadable shows,,,

Today we are happy to roll out a new feature, automatic creation/updating of BitTorrents for downloadable items (shows) in the Live Music Archive.

At the moment, Torrents are only being created when new Items are created (a new show is uploaded) or an existing one is modified.

Items that have Torrents will have a 'Torrent' link (and seed/leech count) next to the familiar HTTP link in the "Play/Download" section at the bottom of the "Listen to audio" box upper right of the Item details page.

We aren't (quite yet) adding Torrents for previously existing shows, but if you want to add a Torrent to an Item in the LMA, just add a review to it (e.g. "Make Torrent :D"). If you have privs (e.g. because you uploaded the Item), make any modification to it (e.g, adding a space in the Description) -- and the system will automagically make a Torrent for you!

We're running two trackers at the moment, but unless your show is popular at a given moment, most likely Torrents will download from our servers via webseeding. So you'll need a BitTorrent client that supports Getright-style webseeds (e.g. Transmission or uTorrent to name two...).

Still, the Item Torrent provides a very easy way to grab an entire Item (show) with a click or two :D.

Enjoy!

-aaron (Internet Archive engineer working on this)

PS Some technical details on our Torrents and our slightly funky use of them here:

http://archive.org/about/faqs.php#Archive_BitTorrents

This post was modified by Aaron Ximm on 2012-07-26 00:04:23

This post was modified by Aaron Ximm on 2012-07-26 00:21:12

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffgreenone Date: Jul 25, 2012 10:01pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Hi Aaron - wow, very cool! I remember at some point last year it was also possible to *upload* a torrent file and have the system pull the shows from whatever tracker the show was on (it was bt.etree.org in this case; not sure what happens with members-only sites). I seem to recall it not working very well - something wrong with the post-download derive process not working, from what I remember - but this jogged my memory. Is that still in development or still possible?

The show I uploaded was here: http://archive.org/details/franti2011-05-22.flac16

I think I deleted the flac files in the meantime hoping to redo it once things got fixed...

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffAaron Ximm Date: Jul 26, 2012 12:27pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

It's still possible to upload torrents, but IIRC the post-upload derive is disable etree (LMA) items (so the torrents aren't retrieved).

The auto-retrieval was turned off within the LMA because there was an existing tradition of some shows being uploaded with the torrents they had originally been captured from included (I was guessing to document provenance).

There were a few cases where the system would automatically retrieved those torrents again into the items, yielding two copies of every 'real' file, and it was confusing to figure out what was going on and what was derived from what!

So at the moment torrents will only get retrieved if they're uploaded to other places. :/

-aaron

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffgreenone Date: Jul 26, 2012 4:52pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Interesting. I can't say I've ever seen anyone upload a torrent file along with the lossless files here, but I certainly haven't seen everything.

I would be beyond happy if that feature were enabled or figured out for etree - bt.etree.org is littered with torrents whose files never make it here for whatever reason. I tried to keep up for a few months earlier this year but got burned out after a while...

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffMonte B Cowboy Date: Jul 31, 2012 9:34am
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

archive.torrent testing yields high d/l bit-rate with my uTorrent client...

screen-capture image of one of my shows
notice that The Archive "hooks into" the same 6969/announce Tracker that etree uses
archive_torrent.jpg

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffMonte B Cowboy Date: Jul 28, 2012 11:56am
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

I just noticed that all the etree items I submitted (except for stream_only) have archive.torrents created for them. Torrents have been created for my new submissions and my old submissions. This includes items I haven't edited in a couple of years. Here's my LMA upload list (excluding stream_only items): twenty etree items with new archive.torrents

All twenty items had their bit torrents uploaded and seeded to bt.etree before they were added to their LMA collections.

Re: "uploading torrent files" - I admit to doing this once, after I first noticed some discussion here about adding torrents. Funny, now this item has both torrents: http://archive.org/details/phil1959-04-05.pitch.dehiss.upgrade.flac

I just did a quick test for one of my shows. I loaded the new archive.torrent URL into my uTorrent client, and I selected several files to d/l. It works very well. My d/l rate was full speed (my ISP speed limit). Nice job all around!

This post was modified by Monte B Cowboy on 2012-07-28 18:56:25

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffbrewster Date: Jul 28, 2012 1:27pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Thank you! Go Aaron.

As for bt.etree.org-- I never visited before, and it is sophisticated and huge. I wonder if we can work better with it.

Greenone-- lets figure how to make this all work.


-brewster


This post was modified by brewster on 2012-07-28 20:27:34

Reply to this post
Reply [edit]

Poster: capnhook Date: Sep 17, 2012 7:32pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Excellent work y'all!

Reply to this post
Reply [edit]

Poster: Brian V. Date: Aug 14, 2012 2:26am
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Thanks all the work done on this.

I haven't read every post on this topic so if this was already discussed, please ignore this post.

My initial thinking was whether the lossless and lossy files would be segregated in the torrent so that it is easy to segregate the flac files. The couple of torrents I tested had all the files listed and it is a bit burdensome to go through and uncheck the lossy files for each track. it is already simple enough to grab just the lossy files with the zip downloads. I was hoping this would make it easier to grab just the lossless files with a one or two click solution. that does not appear to be the case. having different options to grab shows is great. I was just curious if there was any work around for this issue.

I already have downthemall set up on firefox and use free download manager for chrome. both work great. Just wanted to raise the question to see if there is a simple answer.

the process for grabbing the show via torrent worked great though. nice work on this and thanks again.

This post was modified by Brian V. on 2012-08-14 09:26:00

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffAaron Ximm Date: Aug 14, 2012 11:23am
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Hi Brian,

The short answer is, alas, there's no simple solution -- we're using Torrents across our collections and have been wary of any profusion of variations or sub-types. :/

There was some discussion of whether to include 'derivative' file formats (the lossy ones in this case), but we decided it was the straightest path to include everything and let people de-select what they didn't want... then offer multiple flavors of Torrents for each item.

Totally see how especially in this case it'd be super useful to have only FLACs in there!

I wonder if there're Torrent clients that have something akin to downloadthemall's filter-by-file-time options?

-aaron

Reply to this post
Reply [edit]

Poster: Brian V. Date: Aug 14, 2012 5:25pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

Thanks for the response and the work aaron. Like I stated, it is great to have the new option of using torrents.
Deselecting the files is not exactly the end of the world. The torrent I downloaded yesterday has been getting peers all throughout the day, so it seems like people like and are using this option.
Thanks again.

Reply to this post
Reply [edit]

Poster: pir_anha Date: Aug 9, 2012 1:49am
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

hey aaron --

i'm thrilled about archive.org using bittorrent, which is my favourite distribution/storage protocol. it'll also hopefully help with making the mainstream realize that torrent != warez.

your "slightly funky" way means that i can't use your web seeding (but i see why you're using it; i am not complaining about that). but i also won't switch clients, because rtorrent is by far the best for my general use -- and alas there seems to be no indication by the dev that web seeding is in rtorrent's near future. is there anything standing in my way of downloading the torrent files via http, putting them in the correct directory structure, and then seeding using your torrent file? it's a bit cumbersome, but i don't mind doing this with a number of otherwise unseeded torrents, because i want to help with the effort in some small way.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffAaron Ximm Date: Aug 9, 2012 12:09pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

That would be fantastic! And a great service to other rTorrent (etc.) users! :D

I intend to see if I can nudge the rTorrent folks to support webseeding, I have used and like it and was sad we weren't working well with it. When you go down the list of our 'funky' implementation I'm not sure there's anyone who handles them *all*... we're hoping if people value our offerings that might be a bit of incentive to get more uniform adoption of various things!

Thanks for your support!!!
-aaron

Reply to this post
Reply [edit]

Poster: pir_anha Date: Aug 9, 2012 2:14pm
Forum: etree Subject: Re: Internet Archive adds BitTorrents for downloadable shows,,,

oh, nudging them would be great if it came from you guys -- when there has been nudging by others, there's been no love: http://libtorrent.rakshasa.no/ticket/245; other requests were closed with "wontfix".

and yeah crossloading instead of web seeding works quite nicely. hm. i could write a script that handles the download and torrent start automatically, which would make the workaround fairly painless.